Operation display system, operation display apparatus, and operation display program

ABSTRACT

An operation display system having a hardware processor is disclosed. The hardware processor acquires load references serving as a reference of pressing force by which a touch panel receives an operation. The hardware processor compares the acquired load references with each other. The hardware processor controls a degree of a change in pressing force for pressing a touch panel of a second operation display apparatus to be notified when there is a difference as a comparison result of load references of a first operation display apparatus operated by a user most recently and the second operation display apparatus to be subsequently operated among a plurality of operation display apparatuses.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on Japanese Patent Application No. 2015-109135filed on May 28, 2015, the contents of which are incorporated herein byreference.

BACKGROUND

1. Technical Field

The present invention relates to an operation display system, anoperation display apparatus, and an operation display program.

2. Description of Related Arts

In recent years, electronic apparatuses including touch panels, forexample, a cellular phone such as a smart phone and MFP (MultifunctionPeripheral), have been generally spread. The touch panel is normallyconfigured by integrally forming a display device such as a liquidcrystal display with an input device such as a touch pad. For example,when a user inputs information to the touch panel, the user can contactimage objects, such as icons and buttons displayed on a screen of thetouch panel, by using a finger or a touch pen, thereby selecting theaforementioned icons, buttons etc., and inputting the information.Moreover, there has also been developed a touch panel capable ofreceiving various input operations through an action (flickering) oflightly flipping a screen in the state in which the screen has beencontacted with a finger or an action (swiping) of slipping the screen inthe state in which the screen has been contacted with a finger forexample, as well as an action of simply contacting the screen with afinger.

On the other hand, with the high functionality of an electronicapparatus, since information displayed on a touch panel is increased anddiversified, there may be a case in which the size of a screen islimited with only one touch panel and it is not possible to display allnecessary information at a time. In such a case, by scrolling the screenin a vertical or horizontal direction, it is possible to see an unseenpart. However, since the size of a screen visible at a time is limited,apart seen up to now becomes unseen, resulting in inconvenience.

In this regard, there has been known a technology (hereinafter, writtenas “cooperative display”) of allowing a plurality of electronicapparatuses including touch panels to cooperate with one another,thereby configuring one large screen by a plurality of touch panels anddisplaying image objects associated with one another across theplurality of touch panels. When such a technology is used, image objectscan be displayed on a larger screen by using the plurality of touchpanels. Furthermore, in the plurality of touch panels subjected to thecooperative display, a user can perform operations among the pluralityof touch panels (hereinafter, written as a “cooperative operation”)similarly to the case of operating one touch panel.

However, in touch panels of electronic apparatuses currently on themarket, several types exist depending on operational principles, andbased on the types, a difference may exist in references of pressingforce (hereinafter, written as “load references”) by which the touchpanels receive operations. For example, as a main type of a touch panel,there are a resistive-film type and a capacitive type. In aresistive-film type touch panel, a load reference is higher than that ofa capacitive type touch panel, and input pressure needs to be strongerthan that of a capacitive type touch panel. Consequently, for example,when a flick operation or a swipe operation is performed, a user mayfeel that a touch panel with a low load reference is easy to operatebecause it is possible to operate it at small pressure force, but mayfeel that a touch panel with a high load reference is difficult tooperate.

In this regard, there has been known a technology for improving theoperability of a touch panel, for example, a technology of JapaneseUnexamined Publication No. 2011-150737. Japanese Unexamined PublicationNo. 2011-150737 discloses a technology aiming at suppressing an abnormaloperation of the touch panel by allowing load references to differ inadjacent image objects. When such a technology is used, it is possibleto expect the improvement of the operability of individual touch panels.

However, although the operability of individual touch panels has beenimproved using the technology as disclosed in Japanese UnexaminedPublication No. 2011-150737, when a plurality of electronic apparatusesare allowed to cooperate with one another, if there is a difference inthe operability of the respective touch panels, it is difficult to beoperated by a user. For example, when a plurality of electronicapparatuses including touch panels having different load references areallowed to cooperate with one another, there is a difference between theoperability of an electronic apparatus including a touch panel having ahigh load reference and the operability of an electronic apparatusincluding a touch panel having a low load reference. As a consequence,since the touch panels do not operate as the intention of a user, theuser may feel a burden.

For example, in two operation display apparatuses having different loadreferences, when a flick operation is performed on a touch panel havinga low load reference and then is performed on a touch panel having ahigh load reference, a user may feel that the operability of the touchpanels has been suddenly reduced.

Furthermore, when a user performs a multi-touch operation with respectto an operable object displayed across two operation display apparatuseshaving different load references by using his/her thumb, forefinger,etc., since it is necessary to apply different pressure force torespective fingers, a user may feel a burden.

SUMMARY

The present invention is intended to solve the aforementioned problemsin the prior art, and one of the objects of the present invention is toprovide an operation display system, an operation display apparatus, andoperation display program, by which it is possible to reduce a burden ofa user due to a difference in operability caused by a difference in loadreferences between operation display apparatuses.

In order to achieve at least one of the aforementioned objects, anoperation display system having a plurality of operation displayapparatuses including a touch panel, reflecting one aspect of thepresent invention, comprises a hardware processor configured to: acquireload references serving as a reference of pressing force by which thetouch panel receives an operation, compare the acquired load referenceswith each other. The hardware processor controls a degree of a change inpressing force for pressing a touch panel of a second operation displayapparatus to be notified when there is a difference as a comparisonresult of load references of a first operation display apparatusoperated by a user most recently and the second operation displayapparatus to be subsequently operated among the plurality of operationdisplay apparatuses.

Preferably, the hardware processor controls the degree of the change inthe pressing force to be displayed on a screen of any one of theplurality of operation display apparatuses.

Preferably, the hardware processor controls the degree of the change inthe pressing force to be displayed on a screen of the second operationdisplay apparatus.

Preferably, the operation display system further has a line-of-sightdetector for detecting a line-of-sight of a user, and on a basis of adetection result of the line-of-sight detector, the hardware processordisplays the degree of the change in the pressing force on a screen ofan operation display apparatus currently viewed by a user among theplurality of operation display apparatuses.

The objects, features, and characteristics of this invention other thanthose set forth above will become apparent from the description givenherein below with reference to preferred embodiments illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram for explaining a configuration of anoperation display system of a first embodiment of the present invention.

FIG. 2 is a side view for explaining a schematic configuration of animage forming apparatus illustrated in FIG. 1.

FIG. 3 is a block diagram for explaining a schematic configuration of animage forming apparatus illustrated in FIG. 1.

FIG. 4 is a block diagram for explaining a schematic configuration of aportable terminal illustrated in FIG. 1.

FIG. 5 is a schematic diagram for explaining cooperation of an operationdisplay apparatus in an operation display system of a first embodimentof the present invention.

FIG. 6A is a flowchart for explaining one example of a processingprocedure of an operation display method of a first embodiment of thepresent invention.

FIG. 6B is a flowchart subsequent to FIG. 6A.

FIG. 6C is a sub-routine flowchart for explaining a process (step S107)of displaying pressing information of FIG. 6A.

FIG. 6D is a sub-routine flowchart for explaining a process (step S110)of displaying pressing information of FIG. 6B.

FIG. 7 is a schematic diagram illustrating switching of an object to beoperated from an apparatus having a low load reference to an apparatushaving a high load reference in a first embodiment of the presentinvention.

FIG. 8 is a schematic diagram illustrating switching of an object to beoperated from an apparatus having a high load reference to an apparatushaving a low load reference in a first embodiment of the presentinvention.

FIG. 9A is a schematic diagram illustrating one example of visuallyexpressing the degree of a change in pressing force in a firstembodiment of the present invention.

FIG. 9B is a schematic diagram illustrating another example of visuallyexpressing the degree of a change in pressing force in a firstembodiment of the present invention.

FIG. 10A is a flowchart for explaining one example of a processingprocedure of an operation display method in a second embodiment of thepresent invention.

FIG. 10B is a sub-routine flowchart for explaining a process (step S407)of adjusting a position of an operation object of FIG. 10A.

FIG. 10C is a sub-routine flowchart for explaining a process (step S408)of notifying pressing information of FIG. 10A.

FIG. 11 is a schematic diagram for explaining position adjustment of anoperation object in a second embodiment of the present invention.

FIG. 12 is a schematic diagram for explaining notification of pressinginformation in a second embodiment of the present invention.

FIG. 13A is a flowchart for explaining one example of a processingprocedure of an operation display method in a third embodiment of thepresent invention.

FIG. 13B is a sub-routine flowchart for explaining a process (step S708)of notifying pressing information of FIG. 13A.

FIG. 14 is a schematic diagram for explaining notification of pressinginformation in a third embodiment of the present invention.

DETAILED DESCRIPTION

The embodiments of an operation display system, an operation displayapparatus, and an operation display program of this invention will bedescribed below with reference to the accompanying drawings. Inaddition, in a description of the drawings, the same reference numeralsare used to designate the same elements.

First Embodiment

<Configuration of Operation Display System>

FIG. 1 is a schematic diagram for explaining a configuration of anoperation display system of a first embodiment of the present invention.Furthermore, FIG. 2 and FIG. 3 are a side view and a block diagram forexplaining a schematic configuration of an image forming apparatusillustrated in FIG. 1, respectively.

As illustrated in FIG. 1, an operation display system 400 has an imageforming apparatus 100, portable terminal apparatuses 200A, 200B, and200C, and a server 300. These elements are connected to an access pointand are configured to be able to communicate with one another via anetwork N. In addition, the aforementioned elements may also be directlyconnected to one another without using the access point and the network.Furthermore, the aforementioned elements may also be connected to oneanother in a wireless manner, or in a wired manner.

The image forming apparatus 100 of the present embodiment, for example,may be an MFP, a printer, a facsimile, etc. Hereinafter, the case inwhich the image forming apparatus 100 is the MFP will be described as anexample. Furthermore, the portable terminal apparatuses 200A, 200B, and200C, for example, may be a cellular phone such as a smart phone, orportable information terminals such as PDA (personal DigitalAssistants). Hereinafter, in the present specification, for the purposeof convenience, the portable terminal apparatuses are written as“portable terminals”.

In addition, FIG. 1 illustrates the case in which one MFP, threeportable terminals, and one server are connected to the network N;however, the number of MFPs, the number of portable terminals, and thenumber of servers are not limited to this case. Furthermore, any one ofthe aforementioned elements may also be omitted or other elements mayalso be added.

Furthermore, the network N includes various networks such as LAN (LocalArea Network) that connects computers or network devices to one anotherby a standard like Ethernet (registered trademark), Token Ring, FDDI(Fiber-Distributed Data Interface), etc., WAN (Wide Area Network) thatconnects LANs to one another by dedicated lines, the Internet, and acombination thereof. A network protocol, for example, is TCP/IP(Transmission Control Protocol/Internet Protocol).

<Configuration of MFP>

As illustrated in FIG. 2 and FIG. 3, the MFP 100 has an image readingunit 110, an image processing unit 120, an image forming unit 130, acommunication unit 140, a paper feeding unit 150, an operation displayunit 160, an imaging unit 170, and a control unit 180. These elementsare connected to one another by an internal connection line 190. In thepresent embodiment, the MFP 100 serves as an operation displayapparatus.

The image reading unit 110 optically reads a document and generates animage data signal. The image reading unit 110 has a light source, anoptical system, an image sensor, and an analog-to-digital conversioncircuit. In the image reading unit 110, the light source sequentiallyirradiates light of each color (R, G, and B) to the document. Theoptical system has a plurality of mirrors and an image forming lens, andan image of reflected light from the document is formed on the imagesensor through the mirrors and the image forming lens of the opticalsystem. The image sensor reads reflected light corresponding to eachcolor (R, G, and B) in each line, and generates an electrical signal inresponse to the intensity of the reflected light from the document. Thegenerated electrical signal is converted from an analog signal to adigital signal in the analog-to-digital conversion circuit, and istransmitted to the image processing unit 120 as an image data signal.

The image processing unit 120 performs various image processes on theaforementioned image data signal, thereby generating print image data.The image processes, for example, include shading correction, gammacorrection, inter-line correction, color shift correction, anenlargement reduction process, a color conversion process, etc.

The image forming unit 130 forms an image by an electrophotographicprocess on the basis of the print image data, and fixes the formed imageto a paper as a recording medium. In more detail, the image forming unit130 allows a photoreceptor to be charged by a charging device, andapplies light to the aforementioned photoreceptor by using a lightemitting element such as laser and LED (Light Emitting Diode) on thebasis of the print image data, thereby forming an electrostatic latentimage. The print image data is generated in the image processing unit120 or is generated by rasterizing a print job received in thecommunication unit 140. Then, the image forming unit 130 attaches tonerto the electrostatic latent image on the aforementioned photoreceptorand then transfers the toner to a paper. The paper is fed by the paperfeeding unit 150 and is conveyed to the image forming unit 130 by apaper conveying unit. The paper subjected to the transfer is conveyed toa fixing device through a conveying belt, and the fixing device heatsand presses the toner transferred to the paper, thereby adhering thetoner. The paper with an image formed through the adhesion of the toneris discharged to an exterior from a paper discharge tray.

The communication unit 140 is an interface for communicating with otherapparatuses such as the portable terminals 200A, 200B, and 200C and theserver 300 via a network. For the communication, standards, such as LTE(Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access),Ethernet (registered trademark), Wi-Fi, FDDI, and token ring, are used.Furthermore, the communication unit 140, for example, has a near fieldcommunication function by Bluetooth (registered trademark), and cantransmit/receive image data, etc., to/from the portable terminals 200A,200B, and 200C. The communication unit 140 serves as a part of ahardware processor together with the control unit 180.

The paper feeding unit 150 includes at least one paper feeding tray andsupplies papers to the image forming unit 130 one by one.

The operation display unit 160 receives various types of input from auser and displays information such as various types of setting,processing results, alerts, and messages. The operation display unit 160includes a touch panel having a touch pad and a display device, and hasa structure capable of processing the reception of input in parallelwith the display of information. The touch panel can employ any one of aresistive-film type, a capacitive type, a surface acoustic wave type, aninfrared type, an electromagnetic induction type, and a load detectiontype. Hereinafter, the case in which the operation display unit 160includes a resistive-film type touch panel will be described as anexample. Furthermore, the display device is arranged on a back surfaceof the touch pad and may include a liquid crystal display, an organiclight emitting display, etc.

In the case of inputting information to the operation display unit 160,a user can contact with an object, which has been displayed on a screenof the operation display unit 160, by using a finger or a touch pen,thereby selecting the aforementioned object and inputting information.In addition, in the present specification, contacting with a touch panelby a user at pressing force received by the touch panel as input iscalled “touching”. Furthermore, an object, for example, may include anicon, a button, and a background image drawn on the background of theicon or the button. Furthermore, in the present specification, an objectcapable of receiving an input operation is written as an “operationobject”.

Furthermore, a user can input information to the operation display unit160 by using various operations (gestures) of fingers, such as moving afinger on a screen in the state of touching the screen with the finger,as well as an operation for simply touching the screen with a finger andinputting information. The operation display unit 160 of the presentembodiment, for example, can receive tap, double tap, triple tap, longtap, drag, flick, swipe, pinch-in, pinch-out, etc., as input operationswithout particularly limiting thereto. Hereinafter, in the presentspecification, for the purpose of convenience, operations other than thetap, the double tap, the triple tap, and the long tap among theaforementioned input operations, for example, the drag, the flick, theswipe, the pinch-in, the pinch-out, etc., are written as “touchoperations”. Particularly, an operation for simultaneously touching aplurality of places of the touch panel, such as the pinch-in and thepinch-out, is written as a “multi-touch operation”. In general, thetouch operation and the multi-touch operation are input operationsrequiring more complicated operations of fingers as compared with thetap, the double tap, the triple tap, and the long tap.

The imaging unit 170 captures an object on the touch panel of theoperation display unit 160. In the present embodiment, the imaging unit170 includes a camera and serves as a line-of-sight detection unit. Inthe present embodiment, the imaging unit 170 captures eyes of a user inorder to detect a line-of-sight direction of the user. Data captured bythe imaging unit 170 is transmitted to the control unit 180.

The control unit 180 controls the image reading unit 110, the imageprocessing unit 120, the image forming unit 130, the communication unit140, the paper feeding unit 150, the operation display unit 160, and theimaging unit 170.

As illustrated in FIG. 3, the control unit 180 has an auxiliary storagedevice 181, a memory 182, and a CPU 183.

The auxiliary storage device 181, for example, is a large capacitystorage device such as a hard disk drive and a solid state drive. Theauxiliary storage device 181 stores various software programs such as anoperating system and an operation display program P. The operationdisplay program P is a software program for performing an operationdisplay method of the present embodiment as will be described later. Theoperation display program P may also be installed in the auxiliarystorage device 181 in advance, or may also be downloaded from the server300 according to necessity.

The memory 182 includes a memory device such as RAM (Random AccessMemory) and flash ROM (Read Only Memory). The aforementioned varioussoftware programs are temporarily stored in the aforementioned RAM whenthey are executed by the CPU 183. Furthermore, the aforementioned RAMstores a determination result and an arithmetic calculation result bythe CPU 183. Furthermore, the aforementioned flash ROM stores varioustypes of setting such as a load reference of the touch panel of theoperation display unit 160, an operation type of an object, an operationobject display mode, a line-of-sight mode, a notification mode, and anautomatic shift mode.

The load reference is a reference of pressing force for deciding whetherthe touch panel receives an operation, and when the touch panel has beenpressed at pressing force equal to or more than the reference, input isreceived. However, when the touch panel has been pressed at pressingforce smaller than the reference, no input is received. In the presentembodiment, preferably, the load reference has been normalized so as tobe comparable among different types of touch panels in the sameconditions.

The operation type of the object is a type regarding an operation of thetouch panel with respect to each object displayed on the operationdisplay unit 160. The operation type of the object, for example,includes a tap operation, a flick operation, etc. When the flickoperation is included in the operation type of the object, it indicatesthat a corresponding object can be subjected to the flick operation.

The operation object display mode, the line-of-sight mode, thenotification mode, and the automatic shift mode may be set through amanager menu. Details of these modes will be described later.

In the present embodiment, the control unit 180 acquires the loadreferences of touch panels with respect to the MFP 100 and a portableterminal performing cooperative display with the MFP 100, and comparesthe load references with each other. In response to the determinationresult of the load references, the control unit 180 controls the displayof the degree (for example, “strong” or “weak”) of a change frompressing force for a first operation display apparatus operated by auser most recently to pressing force for a second operation displayapparatus to be subsequently operated.

<Configuration of Portable Terminal>

Next, with reference to FIG. 4, the outline of the configuration of eachof the portable terminals 200A, 200B, and 200C will be described. FIG. 4is a block diagram for explaining a schematic configuration of theportable terminal illustrated in FIG. 1. In addition, since theconfigurations of the portable terminals 200A, 200B, and 200C are equalto one another, the configuration of the portable terminal 200A will bedescribed below and a description of the configurations of the portableterminals 200B and 200C will be omitted in order to avoid the redundancyof a description.

The portable terminal 200A has a communication unit 210, an operationdisplay unit 220, an imaging unit 230, and a control unit 240. In thepresent embodiment, the portable terminal 200A serves as an operationdisplay apparatus.

The communication unit 210 is an interface for communicating with otherapparatuses such as the MFP 100 and the server 300 via the network N.For the communication, standards, such as LTE, W-CDMA, Ethernet(registered trademark), Wi-Fi, FDDI, and token ring, are used.Furthermore, the communication unit 210, for example, has a near fieldcommunication function by Bluetooth (registered trademark), and cantransmit/receive image data, etc., to/from the MFP 100.

The communication unit 210 serves as a part of a hardware processortogether with the control unit 240.

The operation display unit 220 receives various types of input from auser and displays information such as various types of setting,processing results, alerts, and messages. The operation display unit 220includes a touch panel having a touch pad and a display device, and hasa structure capable of processing the reception of input in parallelwith the display of information. The touch panel can employ any one of aresistive-film type, a capacitive type, a surface acoustic wave type, aninfrared type, an electromagnetic induction type, and a load detectiontype. Hereinafter, the case in which the portable terminal 200A includesa capacitive type touch panel will be described as an example.Furthermore, the display device is arranged on a back surface of thetouch pad and may include a liquid crystal display, an organic lightemitting display, etc.

The imaging unit 230 takes an image of an object on the touch panel ofthe operation display unit 220. In the present embodiment, the imagingunit 230 includes a camera and serves as a line-of-sight detection unit.In the present embodiment, the imaging unit 230 takes an image of eyesof a user in order to detect a line-of-sight direction of the user. Datacaptured by the imaging unit 230 is transmitted to the control unit 240.

The control unit 240 controls the communication unit 210, the operationdisplay unit 220, and the imaging unit 230. As illustrated in FIG. 4,the control unit 240 has an auxiliary storage device 241, a memory 242,and a CPU 243.

The auxiliary storage device 241 stores various software programs suchas an operating system and an operation display program P′. Theoperation display program P′ is a software program for performing anoperation display method of the present embodiment as will be describedlater. The operation display program P′ may also be installed in theauxiliary storage device 241 in advance, or may also be downloaded fromthe server 300 according to necessity.

Since the memory 242 has a configuration similar to that of the memory182 of the MFP 100, a detailed description will be omitted in order toavoid the redundancy.

<Cooperation of Operation Display Apparatus>

Next, with reference to FIG. 5, the cooperation of the operation displayapparatus of the present embodiment will be described. FIG. 5 is aschematic diagram for explaining the cooperation of the operationdisplay apparatus in the operation display system of the firstembodiment.

In the present embodiment, when a distance between the MFP 100 and theportable terminal 200A is within a predetermined distance, the MFP 100and the portable terminal 200A start to cooperate with each other.

When the cooperation is started, the control unit 180 and the controlunit 240 cooperate with each other, so that display data and a controlsignal are exchanged via the communication unit 140 and thecommunication unit 210. For example, when the portable terminal 200A isplaced at the right of the MFP 100, the control unit 180 and the controlunit 240 control objects associated with each other to be continuouslydisplayed over the operation display unit 220 from the operation displayunit 160. In the example illustrated in FIG. 5, the operation displayunit 160 and the operation display unit 220 configure one large screenand display operation objects (buttons) of a “func. 1” (function 1) to a“func. 4” (function 4) which are a part of a menu screen.

When a user operates the aforementioned menu screen by touching it, theoperation display unit 160 and the operation display unit 220 process ascreen operation of the user in cooperation with each other.Consequently, the user can operate the operation display unit 160 andthe operation display unit 220 as one operation display unit. Forexample, when the user scrolls the display content of the operationdisplay unit 160 in the right direction by a swipe operation, etc., thedisplay content of the operation display unit 220 is also scrolled inthe right direction according to the scrolling. As described above, theoperation display unit 160 and the operation display unit 220 integrallydisplay an object such that the display content of the operation displayunit 160 and the display content of the operation display unit 220 areoperated in response to a screen operation.

In addition, when an operation object (the button of the “function 3” inFIG. 5) displayed across the two touch panels of the operation displayunit 160 and the operation display unit 220 is an object which can besubjected to a multi-operation, a user can also perform themulti-operation with respect to the object.

Furthermore, in FIG. 5, the case in which two operation displayapparatuses of the MFP 100 and the portable terminal 200A cooperate witheach other is illustrated as an example; however, the number ofcooperable operation display apparatuses in the present embodiment isnot limited to two.

Furthermore, the aforementioned distance is not particularly limited;however, preferably, it is set to about several cm. However, inconsideration of the convenience of a user, the aforementioned distancecan also be set to several meters to several tens of meters.

Moreover, in FIG. 5, the case in which two operation display apparatusesof the MFP 100 and the portable terminal 200A cooperate with each otherin the horizontal direction of the paper surface is illustrated as anexample; however, they can also cooperate with each other in thevertical direction.

As described above, with only one touch panel, since the size of ascreen is limited, when there is an object not displayable at a time, aplurality of operation display apparatuses are allowed to cooperate withone another, so that it is possible to display many more objects at atime and operate them.

A detailed example in which a processing procedure of the operationdisplay method of the present embodiment is performed using the MFP 100and the portable terminal 200A having the above configurations will bedescribed with reference to FIG. 6A to FIG. 6D, FIG. 7, and FIG. 8. Theoperation display method of the present embodiment is performed when theaforementioned operation display programs P and P′ are respectivelyexecuted by the CPUs 183 and 243. The operation display programs P andP′ are configured such that the MFP 100 and the portable terminal 200Aperform the operation display method of the present embodiment incooperation with each other. Furthermore, in the operation displayprograms P and P′, the operation display program P may also play aleading role or the operation display program P′ may also play a leadingrole in a relation between these two programs. The following descriptionwill be provided for the case in which the operation display program Pplays a leading role.

<Outline of Operation Display Method>

FIG. 6A is a flowchart for explaining one example of a processingprocedure of an operation display method of the first embodiment, andFIG. 6B is a diagram subsequent to FIG. 6A. Furthermore, FIG. 6C is asub-routine flowchart for explaining a process (step S107) of displayingpressing information of FIG. 6A, and FIG. 6D is a sub-routine flowchartfor explaining a process (step S110) of displaying pressing informationof FIG. 6B. Furthermore, FIG. 7 is a schematic diagram illustratingswitching of an object to be operated from an apparatus having a lowload reference to an apparatus having a high load reference in the firstembodiment. Furthermore, FIG. 8 is a schematic diagram illustratingswitching of an object to be operated from an apparatus having a highload reference to an apparatus having a low load reference in the firstembodiment.

<Main Routine of Operation Display Method>

As illustrated in FIG. 6A, firstly, it is determined whether there is acommunicable portable terminal nearby (step S101). For example, when adistance between the MFP 100 and the portable terminal 200A is withinthe aforementioned predetermined distance, the control unit 180determines that there is the communicable portable terminal nearby (stepS101: YES) and starts cooperative display with the portable terminal200A (step S102). The following description will be provided for thecase in which the MFP 100 and the portable terminal 200A performcooperative display. On the other hand, there is no communicableportable terminal nearby (step S101: NO), the control unit 180 waitsuntil a communicable portable terminal comes nearby.

Next, the control unit 180 acquires load references of the MFP 100 andthe portable terminal 200A (step S103). The control unit 180 reads aload reference N1 of the touch panel of the operation display unit 160from the flash ROM of the memory 182 and achieves it. Furthermore, thecontrol unit 240 reads a load reference N2 of the touch panel of theoperation display unit 220 from the memory 242 and transmits it throughthe communication unit 210. The control unit 180 receives the loadreference N2 of the touch panel of the operation display unit 220through the communication unit 140 and achieves it.

Next, the control unit 180 detects an apparatus to be subsequentlyoperated by a user (step S104). The apparatus to be subsequentlyoperated by the user (the second operation display apparatus) is anapparatus different from an apparatus operated by the user most recently(the first operation display apparatus), and is considered as anapparatus in which a user's finger or a touch pen initially touches atouch panel. In more detail, when the user has touched the touch panel,the operation display unit 160 notifies the control unit 180 of the factthat the user has touched the touch panel. Furthermore, when the userhas touched the touch panel, the operation display unit 220 notifies thecontrol unit 240 of the fact that the user has touched the touch panel.Then, the control unit 240 notifies the control unit 180 of the factthat the user has touched the touch panel of the operation display unit220 through the communication unit 210. As described above, the controlunit 180 detects the apparatus in which the user's finger or the touchpen has initially touched the touch panel, thereby detecting the secondoperation display apparatus.

Next, the control unit 180 determines whether an object to be operatedhas been switched (step S105). When the second operation displayapparatus detected in the aforementioned step S104 is equal to the firstoperation display apparatus, the control unit 180 determines that theobject to be operated has not been switched (step S105: NO) and proceedsto a process of step S111.

On the other hand, when the second operation display apparatus detectedin step S104 is different from the first operation display apparatus,the control unit 180 determines that the object to be operated has beenswitched (step S105: YES).

Next, the control unit 180 determines whether the operation objectdisplay mode is ON (step S106). The control unit 180 reads settinginformation of the operation object display mode from the memory 182,and proceeds to a process of step S108 when the operation object displaymode is not ON (step S106: NO).

On the other hand, when the operation object display mode is ON (stepS106: YES), the control unit 180 displays pressing information (stepS107). The pressing information is information on pressing required foran operation of the touch panel of the second operation displayapparatus. The control unit 180 displays pressing information inresponse to the object to be operated. A detailed process of displayingthe pressing information will be described later.

Next, as illustrated in FIG. 6B, the control unit 180 determines whetherthe line-of-sight mode is ON (step S108). The control unit 180 readssetting information of the line-of-sight from the memory 182, andproceeds to the process of step S111 when the line-of-sight is not ON(step S108: NO).

On the other hand, when the line-of-sight mode is ON (step S108: YES),the control unit 180 detects the line-of-sight direction of the user(step S109). The control unit 180 controls the imaging unit 170 to takean image of the eyes of the user, and estimates the line-of-sightdirection of the user on the basis of the image result. In addition,since an algorithm for estimating the line-of-sight direction on thebasis of the image result of the eyes is well-known, a detaileddescription thereof will be omitted.

Next, the control unit 180 displays pressing information (step S110).When there is the MFP 100 in front of the estimated line-of-sightdirection of the user, the control unit 180 controls the operationdisplay unit 160 to display the pressing information. Alternatively,when there is the portable terminal 200A in front of the estimatedline-of-sight direction of the user, the control unit 180 requests thecontrol unit 240 such that the operation display unit 220 displays thepressing information. A detailed process of displaying the pressinginformation will be described later.

Next, the control unit 180 determines whether the position of a portableterminal to be cooperated has been separated (step S111). When theposition of the portable terminal 200A to be cooperated has not beenseparated beyond the aforementioned predetermined distance with respectto the MFP 100 (step S111: NO), the procedure proceeds to the process ofstep S104.

On the other hand, when the position of the portable terminal to becooperated has been separated (step S111: YES), the control unit 180releases the cooperative display with the portable terminal 200A (stepS112) and ends the procedure (end).

<Process (Step S107) of Displaying Pressing Information>

As illustrated in FIG. 6C, firstly, the control unit 180 determineswhether an object to be operated is the MFP 100 (step S201). When theobject to be cooperated is the MFP 100 (step S201: YES), the controlunit 180 determines whether the load reference N1 is larger than theload reference N2 (step S202). When N1 is larger than N2 (step S202:YES), the control unit 180 controls the operation display unit 160 todisplay strong pressing (step S203). For example, in the exampleillustrated in FIG. 7, a user intends to perform a flick operationtoward the MFP 100 (the load reference N1: high) from the portableterminal 200A (the load reference N2: low). In this case, the controlunit 180 controls a message such as “Press strongly as compared with theright apparatus, please” to be displayed on the screen of the operationdisplay unit 160, thereby promoting the user to strongly operate thetouch panel of the operation display unit 160. Then, the control unit180 ends the process of displaying the pressing information and proceeds(returns) to the process of step S108 of the flowchart of FIG. 6B.

On the other hand, when N1 is not larger than N2 (step S202: NO), thecontrol unit 180 determines whether N1 is equal to N2 (step S204). WhenN1 is equal to N2 (step S204: YES), the control unit 180 ends theprocess of displaying the pressing information and proceeds (returns) tothe process of step S108 of the flowchart of FIG. 6B.

On the other hand, when N1 is not equal to N2 (step S204: NO), that is,when N1 is smaller than N2, the control unit 180 controls the operationdisplay unit 160 to display light pressing (step S205). For example,when a user intends to per forma flick operation toward the MFP 100 (theload reference N1: low) from the portable terminal 200A (the loadreference N2: high), the control unit 180 controls the operation displayunit 160 to display a message such as “Press lightly as compared withthe right apparatus, please”. Then, the control unit 180 ends theprocess of displaying the pressing information and proceeds (returns) tothe process of step S108 of the flowchart of FIG. 6B.

On the other hand, when the object to be cooperated is not the MFP 100(step S201: NO), that is, when the object to be cooperated is theportable terminal 200A, the control unit 180 determines whether N1 islarger than N2 (step S206). When N1 is larger than N2 (step S206: YES),the control unit 240 controls the operation display unit 220 to displaylight pressing (step S207). For example, in the example illustrated inFIG. 8, a user intends to perform a flick operation toward the portableterminal 200A (the load reference N2: low) from the MFP 100 (the loadreference N1: high). In this case, the control unit 240 controls amessage such as “Press lightly to flick as compared with the leftapparatus, please” to be displayed on the screen of the operationdisplay unit 220. Then, the control unit 180 ends the process ofdisplaying the pressing information and proceeds (returns) to theprocess of step S108 of the flowchart of FIG. 6B.

On the other hand, when N1 is not larger than N2 (step S206: NO), thecontrol unit 180 determines whether N1 is equal to N2 (step S208). WhenN1 is equal to N2 (step S208: YES), the control unit 180 ends theprocess of displaying the pressing information and proceeds (returns) tothe process of step S108 of the flowchart of FIG. 6B.

On the other hand, when N1 is not equal to N2 (step S208: NO), that is,when N1 is smaller than N2, the control unit 240 controls the operationdisplay unit 220 to display strong pressing (step S209). For example,when a user intends to perform a flick operation toward the portableterminal 200A (the load reference N2: high) from the MFP 100 (the loadreference N1: low), the control unit 240 controls a message such as“Press lightly as compared with the left apparatus, please” to bedisplayed on the screen of the operation display unit 220. Then, thecontrol unit 180 ends the process of displaying the pressing informationand proceeds (returns) to the process of step S108 of the flowchart ofFIG. 6B.

<Process (Step S110) of Displaying Pressing Information>

As illustrated in FIG. 6D, firstly, the control unit 180 determineswhether a line-of-sight target is the MFP 100 (step S301). When theline-of-sight target is the MFP 100 (step S301: YES), the control unit180 determines whether N1 is larger than N2 (step S302). When N1 islarger than N2 (step S302: YES), the control unit 180 controls theoperation display unit 160 to display strong pressing (step S303). Then,the control unit 180 ends the process of displaying the pressinginformation and proceeds (returns) to the process of step S111 of theflowchart of FIG. 6B.

On the other hand, when N1 is not larger than N2 (step S302: NO), thecontrol unit 180 determines whether N1 is equal to N2 (step S304). WhenN1 is equal to N2 (step S304: YES), the control unit 180 ends theprocess of displaying the pressing information and proceeds (returns) tothe process of step S111 of the flowchart of FIG. 6B.

On the other hand, when N1 is not equal to N2 (step S304: NO), that is,when N1 is smaller than N2, the control unit 180 controls the operationdisplay unit 160 to display light pressing (step S305). Then, thecontrol unit 180 ends the process of displaying the pressing informationand proceeds (returns) to the process of step S111 of the flowchart ofFIG. 6B.

On the other hand, when the line-of-sight target is not the MFP 100(step S301: NO), that is, when the line-of-sight target is the portableterminal 200A, the control unit 180 determines whether N1 is larger thanN2 (step S306). When N1 is larger than N2 (step S306: YES), the controlunit 240 controls the operation display unit 220 to display lightpressing (step S307).

On the other hand, when N1 is not larger than N2 (step S306: NO), thecontrol unit 180 determines whether N1 is equal to N2 (step S308). WhenN1 is equal to N2 (step S308: YES), the control unit 180 ends theprocess of displaying the pressing information and proceeds (returns) tothe process of step S111 of the flowchart of FIG. 6B.

On the other hand, when N1 is not equal to N2 (step S308: NO), that is,when N1 is smaller than N2, the control unit 240 controls the operationdisplay unit 220 to display strong pressing (step S309). Then, thecontrol unit 180 ends the process of displaying the pressing informationand proceeds (returns) to the process of step S111 of the flowchart ofFIG. 6B.

As described above, one example of the processing procedure of theoperation display method of the present embodiment has been describedwith reference to FIG. 6A to FIG. 6D, FIG. 7, and FIG. 8. In theaforementioned operation display method, the control unit 180 acquiresload references of an apparatus operated by a user most recently and anapparatus to be subsequently operated among the MFP 100 and the portableterminal apparatuses 200A, 200B, and 200C, and compares the loadreferences with each other. Then, as a result obtained by comparing theload references with each other, when there is a difference in the loadreferences, the control unit 180 controls the degree of a change inpressing force for pressing the touch panel of the aforementionedapparatus to be subsequently operated to be displayed.

(Modification)

In one example of the above-described operation display method, the casehas been described in which the degree of a change in the pressing forcefor pressing the touch panel of the operation display apparatus to besubsequently operated is notified to the operation display unit 160 orthe operation display unit 220 by using a language of “strong” or“weak”. However, the degree of a change in the aforementioned pressingforce can also be displayed using other expressing methods withoutnotifying it by using a language. Hereinafter, with reference to FIG. 9Aand FIG. 9B, other examples of notifying the degree of a change in theaforementioned pressing force will be described.

FIG. 9A is a schematic diagram illustrating one example of visuallyexpressing the degree of a change in pressing force in the firstembodiment, and FIG. 9B is a schematic diagram illustrating anotherexample of visually expressing the degree of a change in pressing force.

As illustrated in FIG. 9A, an operation object (for example, a button)to be displayed is expressed with a hard image such as an “unevenstone”, so that it is possible to notify a user that it is necessary tostrongly operate a touch panel with a high load reference.

On the other hand, as illustrated in FIG. 9B, a button is expressed witha smooth (soft) image, so that it is possible to notify a user that itis necessary to lightly operate a touch panel with a low load reference.

Furthermore, instead of notifying the degree of a change in pressingforce by displaying a language or an image on the operation display unit160 or the operation display unit 220, the degree of a change inpressing force may also be notified to a user through sound or voice byusing a speaker embedded in the MFP 100 or the portable terminal 200A.

The operation display system, the operation display apparatus, and theoperation display program of the present embodiment as described abovehave the following effects.

As a comparison result of the load references of the first operationdisplay apparatus operated by a user most recently and the secondoperation display apparatus to be subsequently operated, when there is adifference between them, the degree of a change in pressing force forpressing the touch panel of the aforementioned second operation displayapparatus is notified. Consequently, a difference between the loadreferences of the first operation display apparatus and the secondoperation display apparatus is recognized by a user, so that it ispossible to reduce a burden of the user due to a difference ofoperability caused by the difference between the load references of thefirst operation display apparatus and the second operation displayapparatus.

In detail, when an object to be operated is switched from an apparatuswith a weak load reference to an apparatus with a strong load reference,a user is notified of the fact that strongly pressing is advantageous,so that it is possible to prevent the user from lightly pressing a touchpanel of the apparatus with the strong load reference similarly to theapparatus with the weak load reference. Consequently, the touchoperation of the user is prevented from not being recognized as anoperation, which is caused when the user lightly presses the touch panelof the apparatus with the strong load reference.

Furthermore, since the apparatus with the weak load reference can beoperated at pressing force weaker than that of the apparatus with thestrong load reference, a user normally feels that it is easy to operatein many cases. However, when a user strongly presses the touch panel ofthe apparatus with the weak load reference, the advantage of theapparatus with the weak load reference that it is easy to operate is notutilized, and it is also probable that the touch panel will be damaged.In the present embodiment, when an object to be operated is switchedfrom the apparatus with the strong load reference to the apparatus withthe weak load reference, a user is notified of the fact that lightlypressing is advantageous, so that it is possible to prevent the userfrom strongly pressing the touch panel of the apparatus with the weakload reference similarly to the apparatus with the strong loadreference.

Furthermore, only when there is a difference in load references, thatis, only when notification is required, a user is notified of the degreeof a change in the load references, that is, “strong” when a low loadreference is changed to a high load reference, or “weak” when a highload reference is changed to a low load reference. Consequently, a usercan know pressing information at an optimal timing and can accuratelydetermine whether to press a touch panel at strong pressing force orweak pressing force as compared with a load reference up to now.

Second Embodiment

In the first embodiment, the case has been described in which the degreeof a change in pressing force for pressing touch panels betweenoperation display apparatuses with different load references isnotified.

In a second embodiment, the following description will be provided forthe case in which as a result of a screen operation by a user betweenoperation display apparatuses with different load references, when anoperation object has been displayed across two operation displayapparatuses, the operation object is moved to an apparatus with a weakload reference. Moreover, the following description will be provided forthe case of notifying the degree of a change in pressing force forpressing touch panels. Hereinafter, in order to avoid a redundancy, adescription of a configuration equal to that of the first embodimentwill be omitted.

FIG. 10A is a flowchart for explaining one example of a processingprocedure of an operation display method in the second embodiment.Furthermore, FIG. 10B is a sub-routine flowchart for explaining aprocess (step S407) of adjusting a position of an operation object ofFIG. 10A, and FIG. 10C is a sub-routine flowchart for explaining aprocess (step S408) of notifying pressing information of FIG. 10A.Furthermore, FIG. 11 is a schematic diagram for explaining positionadjustment of an operation object in the second embodiment, and FIG. 12is a schematic diagram for explaining notification of pressinginformation in the second embodiment.

<Main Routine of Operation Display Method>

As illustrated in FIG. 10A, firstly, the control unit 180 determineswhether there is a communicable portable terminal nearby (step S401).When a distance between the MFP 100 and the portable terminal 200A iswithin the aforementioned predetermined distance, the control unit 180determines that there is a communicable portable terminal nearby (stepS401: YES) and starts cooperative display with the portable terminal200A (step S402). The following description will be provided for thecase in which the MFP 100 and the portable terminal 200A performcooperative display. On the other hand, when there is no communicableportable terminal nearby (step S401: NO), the control unit 180 waitsuntil a communicable portable terminal comes nearby.

Next, the control unit 180 determines whether a user has operated ascreen (step S403). The control unit 180 determines whether the user hasoperated a screen of the operation display unit 160. Furthermore, thecontrol unit 240 determines whether the user has operated a screen ofthe operation display unit 220. When the user has operated at least oneof the screens of the operation display unit 160 and the operationdisplay unit 220 (step S403: YES), the screen display of the MFP 100 andthe portable terminal 200A are updated (step S404).

Next, the control unit 180 determines whether an operation object hasbeen displayed across the screens (step S405). When the operation objecthas not been displayed across the screens of the MFP 100 and theportable terminal 200A (step S405: NO), the procedure proceeds to aprocess of step S409.

On the other hand, when the operation object has been displayed acrossthe screens of the MFP 100 and the portable terminal 200A (step S405:YES), load references N1 and N2 of the MFP 100 and the portable terminal200A are acquired (step S406). Since an acquisition method of the loadreferences N1 and N2 is equal to that of the first embodiment, adescription thereof will be omitted.

Next, the position of the operation object is adjusted (step S407). Inresponse to a comparison result of the load references N1 and N2, thecontrol unit 180 moves the operation object displayed across the screensof the MFP 100 and the portable terminal 200A such that the operationobject is within a screen of an apparatus with a weak load reference. Adetailed process of adjusting the position of the operation object willbe described later.

Next, the control unit 180 notifies pressing information (step S408). Inresponse to the comparison result of the load references N1 and N2, thecontrol unit 180 allows the operation display unit 160 or the operationdisplay unit 220 to display the pressing information. A detailed processof notifying the pressing information will be described later.

Next, the control unit 180 determines whether the position of theportable terminal 200A to be cooperated has been separated (step S409).When the position of the portable terminal 200A to be cooperated has notbeen separated beyond the aforementioned predetermined distance (stepS409: NO), the procedure proceeds to the process of step S403.

On the other hand, when the position of the portable terminal 200A to becooperated has been separated (step S409: YES), the control unit 180releases the cooperative display with the portable terminal 200A (stepS410) and ends the procedure (end).

<Process (Step S407) of Adjusting Position of Operation Object>

As illustrated in FIG. 10B, the control unit 180 determines whether anautomatic shift mode is ON (step S501). The control unit 180 readssetting information of the automatic shift mode from the memory 182, andwhen the automatic shift mode is not ON (step S501: NO), the controlunit 180 ends the process of the position adjustment of the operationobject and proceeds (returns) to the process of step S408 of FIG. 10A.

On the other hand, when the automatic shift mode is ON (step S501: YES),the control unit 180 determines whether the load reference N1 is largerthan the load reference N2 (step S502). When N1 is larger than N2 (stepS502: YES), the control unit 180, for example, moves (shifts) anoperation object (an icon) of a “tree image” such that the operationobject is within the screen of the portable terminal 200A (the loadreference N2: low) as illustrated in FIG. 11 (step S503).

On the other hand, when N1 is not larger than N2 (step S502: NO), thecontrol unit 180 determines whether N1 is equal to N2 (step S504). WhenN1 is equal to N2 (step S504: YES), the control unit 180 ends theprocess of the position adjustment of the operation object and proceeds(returns) to the process of step S408 of FIG. 10A.

On the other hand, when N1 is not equal to N2 (step S504: NO), that is,when N1 is smaller than N2, the control unit 180 moves (shifts) theoperation object to be within the screen of the MFP 100 (the loadreference N1: low) (step S505). Then, the control unit 180 ends theprocess of the position adjustment of the operation object and proceeds(returns) to the process of step S408 of FIG. 10A.

<Process (Step S408) of Notifying Pressing Information>

As illustrated in FIG. 10C, the control unit 180 determines whether anotification mode is ON (step S601). The control unit 180 reads settinginformation of the notification mode from the memory 182, and when thenotification mode is not ON (step S601: NO), the control unit 180 endsthe process of notifying the pressing information and proceeds (returns)to the process of step S409 of FIG. 10A.

On the other hand, when the notification mode is ON (step S601: YES),the control unit 180 determines whether the load reference N1 is largerthan the load reference N2 (step S602). When N1 is larger than N2 (stepS602: YES), the control unit 180 controls the operation display unit 160to display strong pressing (step S603). For example, in an exampleillustrated in FIG. 12, as a result obtained when a user has performed aflick operation toward the portable terminal 200A (the load referenceN2: low) from the MFP 100 (the load reference N1: high), the operationobject of the “tree image” has been displayed across the MFP 100 and theportable terminal 200A. In this case, the control unit 180 allows amessage such as “Press strongly as compared with the left apparatus,please” to be displayed on the screen of the operation display unit 160,thereby promoting a user to strongly press the touch panel of theoperation display unit 160. Then, the control unit 180 ends the processof notifying the pressing information and proceeds (returns) to theprocess of step S409 of FIG. 10A.

On the other hand, when N1 is not larger than N2 (step S602: NO), thecontrol unit 180 determines whether N1 is equal to N2 (step S604). WhenN1 is equal to N2 (step S604: YES), the control unit 180 ends theprocess of notifying the pressing information and proceeds (returns) tothe process of step S409 of FIG. 10A.

On the other hand, when N1 is not equal to N2 (step S604: NO), that is,when N1 is smaller than N2, the control unit 240 controls the operationdisplay unit 220 to display strong pressing (step S605). Then, thecontrol unit 180 ends the process of notifying the pressing informationand proceeds (returns) to the process of step S409 of FIG. 10A.

As described above, with reference to FIG. 10A to FIG. 10C, FIG. 11, andFIG. 12, one example of the processing procedure of the operationdisplay method of the present embodiment has been described. In theaforementioned operation display method, the control unit 180 acquiresthe load references of the MFP 100 (a self-apparatus) and the portableterminal 200A (another operation display apparatus) and compares theload references with each other. Then, when an operation object has beendisplayed across the screen of the MFP 100 and the screen of theportable terminal 200A, the control unit 180 changes display control inresponse to a result obtained by comparing the load reference N1 of theMFP 100 with the load reference N2 of the portable terminal 200A. Indetail, the control unit 180 moves the operation object so as to bewithin the screen of an apparatus with a weak load reference. Moreover,the control unit 180 controls the operation display unit 160 or theoperation display unit 220 to notify the degree of a change in pressingforce for pressing a touch panel.

The operation display system, the operation display apparatus, and theoperation display program of the present embodiment as described abovehave the following effects.

When an operation object has been displayed across operation displayapparatuses with different load references, the operation object ismoved so as to be within the screen of an apparatus with a weak loadreference, so that it is easy to perform a multi-touch operation such aspinch-in and pinch-out. Furthermore, a difference in the load referencesof the operation display apparatuses is recognized by a user, so that itis possible to reduce a burden of the user due to a difference ofoperability caused by the difference between the load references.

Third Embodiment

In the second embodiment, the case has been described in which as aresult of a screen operation by a user between operation displayapparatuses with different load references, when an operation object hasbeen displayed across them, the operation object is moved to anapparatus with a weak load reference. Furthermore, the case in which thedegree of a change in pressing force for pressing a touch panel isnotified has been described.

In a third embodiment, the following description will be provided forthe case in which when an operation object has been displayed acrossoperation display apparatuses with different load references,information on an operation of a touch panel is notified if a finger,etc., of a user have touched an operation object of each operationdisplay apparatus. Moreover, the following description will be providedfor the case in which the operation object is moved to an apparatus witha weak load reference when the finger, etc., of the user have touchedthe operation object of each operation display apparatus. Hereinafter,in order to avoid a redundancy, a description of a configuration equalto those of the first and second embodiments will be omitted.

FIG. 13A is a flowchart for explaining one example of a processingprocedure of an operation display method in the third embodiment.Furthermore, FIG. 13B is a sub-routine flowchart for explaining aprocess (step S708) of notifying pressing information of FIG. 13A.Furthermore, FIG. 14 is a schematic diagram for explaining ofnotification of the pressing information in the third embodiment.

<Main Routine of Operation Display Method>

Processes of steps S701 to S706 in FIG. 13A are equal to those of stepsS401 to S406 in FIG. 10A of the second embodiment. Consequently, inorder to avoid a redundancy, a detailed description of the processes ofsteps S701 to S706 will be omitted.

In step S707, in the state in which an operation object has beendisplayed across the operation display apparatuses, it is determinedwhether a finger, etc., of a user have touched an operation object ofeach apparatus. The control unit 180, for example, determines whether athumb and a forefinger of a user have touched operation objects of theMFP 100 and the portable terminal 200A, respectively. When the finger,etc., of the user have not touched the operation object of eachapparatus (step S707: NO), the procedure proceeds to the process of stepS703.

On the other hand, when the finger, etc., of the user have touched theoperation object of each apparatus (step S707: YES), the control unit180 notifies pressing information (step S708). In response to acomparison result of the load references N1 and N2, the control unit 180notifies the pressing information and information on the operation ofthe touch panel. A detailed process of notifying the pressinginformation will be described later.

Next, the position of the operation object is adjusted (step S709).Since the process of the position adjustment of the operation object issimilar to the process illustrated in FIG. 10B of the second embodiment,a detailed description will be omitted.

Next, the control unit 180 determines whether the position of theportable terminal 200A to be cooperated has been separated (step S710).When the position of the portable terminal 200A to be cooperated has notbeen separated beyond the aforementioned predetermined distance (stepS710: NO), the control unit 180 proceeds to the process of step S703.

On the other hand, when the position of the portable terminal 200A to becooperated has been separated (step S710: YES), the control unit 180releases the cooperative display with the portable terminal 200A (stepS711) and ends the procedure (end).

<Process (Step S708) of Notifying Pressing Information>

As illustrated in FIG. 13B, the control unit 180 determines whether anotification mode is ON (step S801). The control unit 180 reads settinginformation of the notification mode from the memory 182, and when thenotification mode is not ON (step S801: NO), the control unit 180 endsthe process of notifying the pressing information and proceeds (returns)to the process of step S709 of FIG. 13A.

On the other hand, when the notification mode is ON (step S801: YES),the control unit 180 determines whether the load reference N1 is largerthan the load reference N2 (step S802). When N1 is larger than N2 (stepS802: YES), the control unit 180 controls the operation display unit 160to display strongly pressing the vicinity touched by a finger (stepS803). Furthermore, the control unit 240 controls the operation displayunit 220 to display lightly pressing the vicinity touched by a finger(step S804). Moreover, the control unit 180 controls the operationdisplay unit 160 to display the fact that an operation is facilitatedwhen the operation object is moved to the portable terminal 200A side(step S805). For example, as illustrated in FIG. 14, when a user's thumbhas touched the portable terminal 200A and the user's forefinger hastouched the MFP 100, the control unit 180 controls messages such as“strongly” and “lightly” to be displayed in the vicinity of theforefinger and the thumb, respectively. Furthermore, the control unit180, for example, controls the operation display unit 160 to display amessage such as “Operation is facilitated when operation object isshifted to the left apparatus” in the vicinity of a part touched by afinger. The aforementioned message may also use a pop-up or may also bea character string. Then, the control unit 180 ends the process ofnotifying the pressing information and proceeds (returns) to the processof step S709 of FIG. 13A.

On the other hand, when N1 is not larger than N2 (step S802: NO), thecontrol unit 180 determines whether N1 is equal to N2 (step S806). WhenN1 is equal to N2 (step S806: YES), the control unit 180 ends theprocess of notifying the pressing information and proceeds (returns) tothe process of step S709 of FIG. 13A.

On the other hand, when N1 is not equal to N2 (step S806: NO), that is,when N1 is smaller than N2, the control unit 180 controls the operationdisplay unit 160 to display lightly pressing the vicinity touched by afinger (step S807). Furthermore, the control unit 240 controls theoperation display unit 220 to display strongly pressing the vicinitytouched by a finger (step S808). Moreover, the control unit 180 controlsthe operation display unit 220 to display the fact that an operation isfacilitated when the operation object is moved to the MFP 100 side.Then, the control unit 180 ends the process of notifying the pressinginformation and proceeds (returns) to the process of step S709 of FIG.13A.

As described above, with reference to FIG. 13A, FIG. 13B, and FIG. 14,one example of the processing procedure of the operation display methodof the present embodiment has been described. In the aforementionedoperation display method, the control unit 180 acquires the loadreferences of the MFP 100 (a self-apparatus) and the portable terminal200A (another operation display apparatus) and compares the loadreferences with each other. When an operation object has been displayedacross the screen of the MFP 100 and the screen of the portable terminal200A and a finger, etc., of a user have touched the operation object ofeach operation display apparatus, pressing information and informationon an operation of a touch panel are notified in response to acomparison result of load references. Moreover, the operation object ismoved so as to be within the screen of an apparatus with a weak loadreference.

The operation display system, the operation display apparatus, and theoperation display program of the present embodiment as described abovehave the following effects.

Information on an operation of a touch panel is notified, so that it ispossible to reduce a burden of a user due to a difference of operabilitycaused by the difference in the load references of operation displayapparatuses. Furthermore, an operation object is moved so as to bewithin the screen of an apparatus with a weak load reference, so that itis easy to perform a multi-touch operation such as pinch-in andpinch-out.

The present invention is not limited only to the above-describedembodiments and various modifications can be made within the scope ofthe appended claims. For example, in the above-described firstembodiment, the case has been described in which an apparatus, which isdifferent from an apparatus operated by a user most recently and has atouch panel to be initially touched by a user's finger or a touch pen,is employed as an apparatus to be subsequently operated by the user.However, the present invention is not limited to such a case, and it isalso possible to take an image of a user's finger or a touch penapproaching an operation display unit by using an imaging unit and todetect an apparatus intended to be subsequently operated by a user byimage analysis on the basis of the image result. According to such aconfiguration, a control unit can allow pressing information to bedisplayed on the operation display unit before a user touches a touchpanel. As a consequence, a user can confirm the pressing information andthen start a touch operation with a sufficient temporal margin, so thatit is possible to reduce a burden of the user due to a difference ofoperability caused by a difference of load references.

Furthermore, in the above-described first to third embodiments, the casehas been described in which the MFP and the portable terminal perform anoperation display method in cooperation with each other and the MFPplays a leading role in a relation between the MFP and the portableterminal. However, the present invention is not limited to such a case,and for example, it may be possible to employ a configuration in which aserver, the MFP and the portable terminal perform an operation displaymethod in cooperation with one another and the server plays a leadingrole in a relation among the server, the MFP, and the portable terminal.That is, the server may also be configured to have a control function ofacquiring load references to be compared and allowing pressinginformation to be displayed on the operation display units of the MFPand the portable terminal.

Furthermore, in the above-described first to third embodiments, the casehas been described in which there is a difference between loadreferences of capacitive type and resistive-film type touch panels.However, the present invention is not limited to such a case, and canalso be applied to the case in which there is a difference between loadreferences of capacitive type touch panels, resistive-film type touchpanels, or other types of touch panels.

Furthermore, in the above-described first to third embodiments, the casehas been described in which a capacitive type touch panel is easy tooperate as compared with a resistive-film type touch panel. However,there may be a case in which a resistive-film type touch panel is easyto operate by peripheral environments and use conditions of a touchpanel. For example, when a user performs an operation while wearing aglove or performs an operation with a dry finger, it is possible topromote the use of an operation display apparatus including aresistive-film type touch panel as compared with a capacitive type touchpanel.

Furthermore, in the above-described first to third embodiments, the casehas been described in which two operation display apparatuses withdifference load references perform cooperative display. However, thepresent invention is not limited to such a case, and can also be appliedto the case in which operation display apparatuses do not performcooperative display.

What is claimed is:
 1. An operation display system comprising: aplurality of operation display apparatuses each including a touch panel;and a hardware processor configured to: acquire load references of therespective touch panels of the plurality of operation displayapparatuses, the load references serving as a reference of pressingforce by which the respective corresponding touch panels receive anoperation, compare the acquired load references of a first operationdisplay apparatus and a second operation display apparatus among theplurality of operation display apparatuses, with each other, and in acase in which it is determined based on a result of the comparison thatthere is a difference between the load references, control a degree of achange in pressing force for pressing the touch panel of the secondoperation display apparatus as compared to the pressing force forpressing the touch panel of the first operation display apparatus, to beindicated in an output notification, the first operation displayapparatus being an operation display apparatus operated by a user mostrecently among the plurality of operation display apparatuses and thesecond operation display apparatus being an operation display apparatusto be subsequently operated among the plurality of operation displayapparatuses.
 2. The operation display system as claimed in claim 1,wherein the hardware processor controls the degree of the change in thepressing force to be displayed on a screen of any one of the pluralityof operation display apparatuses, to thereby output the notification. 3.The operation display system as claimed in claim 2, wherein the hardwareprocessor controls the degree of the change in the pressing force to bedisplayed on a screen of the second operation display apparatus, tothereby output the notification.
 4. The operation display system asclaimed in claim 2, further comprising: a line-of-sight detector fordetecting a line-of-sight of a user, wherein, on a basis of a detectionresult of the line-of-sight detector, the hardware processor displaysthe degree of the change in the pressing force on a screen of anoperation display apparatus currently viewed by a user among theplurality of operation display apparatuses, to thereby output thenotification.
 5. The operation display system as claimed in claim 1,wherein, in response to a result of the comparison indicating that thereis a difference between the load references, the hardware processorcontrols a degree of pressing force for pressing the touch panels of thefirst and second operation display apparatuses to be indicated in theoutput notification.
 6. The operation display system as claimed in claim1, wherein the hardware processor is included in the first operationdisplay apparatus or the second operation display apparatus.
 7. Theoperation display system as claimed in claim 1, wherein the firstoperation display apparatus performs cooperative display with the secondoperation display apparatus.
 8. The operation display system as claimedin claim 1, wherein the load references serve as a reference of aminimum pressing force that is necessary for deciding that therespective corresponding touch panels receive an operation, wherein thehardware processor determines that an operation is received via any oneof the touch panels only in a case in which a pressing force on said anyone of the touch panels is equal to or greater than the correspondingload reference of said any one of the touch panels, wherein the loadreferences are stored in advance in a memory of the correspondingoperation display apparatus, and wherein the hardware processor acquiresthe load references from the corresponding memories.
 9. An operationdisplay system comprising: a plurality of operation display apparatuseseach including a touch panel; and a hardware processor configured to:acquire load references of the respective touch panels of the pluralityof operation display apparatuses, the load references serving as areference of a minimum pressing force that is necessary for decidingthat the respective corresponding touch panels receive an inputoperation, wherein the hardware processor determines that an inputoperation is received via any one of the touch panels only in a case inwhich a pressing force on said any one of the touch panels is equal toor greater than the corresponding load reference of said any one of thetouch panels, wherein the load references are stored in advance in amemory of the corresponding operation display apparatus, and wherein thehardware processor acquires the load references from the correspondingmemories, compare the acquired load references of a first operationdisplay apparatus and a second operation display apparatus among theplurality of operation display apparatuses, with each other, and controldisplay control for displaying an operation object to be changed inresponse to a result of the comparison, when the operation object hasbeen displayed across a screen of the first operation display apparatusand a screen of the second operation display apparatus.
 10. Theoperation display system as claimed in claim 9, wherein, in response toa result of the comparison, the hardware processor controls theoperation object to be moved for display such that an entirety of theoperation object is viewed on a screen of an operation display apparatuswith a weaker load reference among the first operation display apparatusand the second operation display apparatus.
 11. The operation displaysystem as claimed in claim 9, wherein, in response to a result of thecomparison indicating that there is a difference between the acquiredload references, the hardware processor controls a degree of pressingforce for pressing the touch panels of the first and second operationdisplay apparatuses to be indicated in an output notification.
 12. Theoperation display system as claimed in claim 9, wherein the hardwareprocessor is included in the first operation display apparatus or thesecond operation display apparatus.
 13. The operation display system asclaimed in claim 9, wherein the first operation display apparatusperforms cooperative display with the second operation displayapparatus.
 14. An operation display system comprising: a plurality ofoperation display apparatuses each including a touch panel; and ahardware processor configured to: acquire load references of therespective touch panels of the plurality of operation displayapparatuses, the load references serving as a reference of a minimumpressing force that is necessary for deciding that the respectivecorresponding touch panels receive an input operation, wherein thehardware processor determines that an input operation is received viaany one of the touch panels only in a case in which a pressing force onsaid any one of the touch panels is equal to or greater than thecorresponding load reference of said any one of the touch panels,wherein the load references are stored in advance in a memory of thecorresponding operation display apparatus, and wherein the hardwareprocessor acquires the load references from the corresponding memories,compare the acquired load references of a first operation displayapparatus and a second operation display apparatus among the pluralityof operation display apparatuses, with each other, and controlinformation on an operation of at least one of the touch panels to beindicated in an output notification, in response to a result of thecomparison, in a state in which an operation object has been displayedacross a screen of the first operation display apparatus and a screen ofthe second operation display apparatus, when the operation object ofeach screen has been touched.
 15. The operation display system asclaimed in claim 14, wherein, in response to a result of the comparison,the hardware processor controls the operation object to be moved fordisplay such that an entirety of the operation object is viewed on ascreen of an operation display apparatus with a weaker load referenceamong the first operation display apparatus and the second operationdisplay apparatus.
 16. The operation display system as claimed in claim14, wherein, in response to a result indicating that there is adifference between the acquired load references, the hardware processorcontrols a degree of pressing force for pressing touch panels of thefirst and second operation display apparatuses to be indicated in theoutput notification.
 17. The operation display system as claimed inclaim 14, wherein the hardware processor is included in the firstoperation display apparatus or the second operation display apparatus.18. The operation display system as claimed in claim 14, wherein thefirst operation display apparatus performs cooperative display with thesecond operation display apparatus.
 19. An operation display apparatuscomprising: a touch panel; and a hardware processor configured to:acquire a first load reference serving as a reference of pressing forceby which the touch panel receives an operation and a second loadreference serving as a reference of pressing force by which a touchpanel of another operation display apparatus operated by a user mostrecently receives an operation, compare the first load reference and thesecond load reference with each other, and in a case in which it isdetermined based on a result of the comparison that there is adifference between the first load reference and the second loadreference and it is determined that the user intends to subsequentlyoperate the operation display apparatus after having most recentlyoperated said another operation display apparatus, control a degree of achange in pressing force for pressing the touch panel of the operationdisplay apparatus to be indicated in an output notification.
 20. Anoperation display apparatus comprising: a touch panel; and a hardwareprocessor configured to: acquire a first load reference serving as areference of a minimum pressing force that is necessary for decidingthat the touch panel receives an input operation and a second loadreference serving as a reference of a minimum pressing force that isnecessary for deciding that a touch panel of another operation displayapparatus receives an input operation, wherein the hardware processordetermines that an input operation is received via the touch panel ofthe operation display apparatus only in a case in which a pressing forceon the touch panel of the operation display apparatus is equal to orgreater than the first load reference, wherein the first load referenceis stored in advance in a memory of the operation display apparatus, andthe second load reference is stored in advance in a memory of saidanother operation display apparatus, and wherein the hardware processoracquires the first load reference and the second load reference from thecorresponding memories; compare the first load reference and the secondload reference with each other, and change display control fordisplaying an operation object, in response to a result of thecomparison, when the operation object has been displayed across a screenof the operation display apparatus and a screen of said anotheroperation display apparatus.
 21. An operation display apparatuscomprising: a touch panel; and a hardware processor configured to:acquire a first load reference serving as a reference of a minimumpressing force that is necessary for deciding that the touch panelreceives an input operation and a second load reference serving as areference of a minimum pressing force that is necessary for decidingthat a touch panel of another operation display apparatus receives aninput operation, wherein the hardware processor determines that an inputoperation is received via the touch panel of the operation displayapparatus only in a case in which a pressing force on the touch panel ofthe operation display apparatus is equal to or greater than the firstload reference, wherein the first load reference is stored in advance ina memory of the operation display apparatus, and the second loadreference is stored in advance in a memory of said another operationdisplay apparatus, and wherein the hardware processor acquires the firstload reference and the second load reference from the correspondingmemories, compare the first load reference and the second load referencewith each other, and control information on an operation of at least oneof the touch panels to be indicated in an output notification, inresponse to a result of the comparison, in a state in which an operationobject has been displayed across a screen of the operation displayapparatus and a screen of said another operation display apparatus, whenthe operation object of each screen has been touched.
 22. Anon-transitory computer readable recording medium storing an operationdisplay program for controlling an operation display system having aplurality of operation display apparatuses each including a touch panel,wherein the operation display program causes a computer to perform: aprocess of acquiring load references of the respective touch panels of afirst operation display apparatus and a second operation displayapparatus among the plurality of operation display apparatuses, thefirst operation display apparatus being an operation display apparatusoperated by a user most recently among the plurality of operationdisplay apparatuses and the second operation display apparatus being anoperation display apparatus to be subsequently operated among theplurality of operation display apparatuses, the load references servingas a reference of a minimum pressing force that is necessary fordeciding that the respective corresponding touch panels receive an inputoperation, wherein it is determined that an input operation is receivedvia any one of the touch panels only in a case in which a pressing forceon said any one of the touch panels is equal to or greater than thecorresponding load reference of said any one of the touch panels,wherein the load references are respectively stored in advance in amemory of the corresponding operation display apparatus, and wherein theload references are acquired from the corresponding memories; a processof comparing the load reference of the first operation display apparatuswith the load reference of the second operation display apparatus; and aprocess of, when a result of the comparison indicates that there is adifference between the load references of the first and second operationdisplay apparatuses, outputting a notification indicating a degree of achange in pressing force for pressing the touch panel of the secondoperation display apparatus.
 23. A non-transitory computer readablerecording medium storing an operation display program for controlling anoperation display system having a plurality of operation displayapparatuses each including a touch panel, wherein the operation displayprogram is executable by a computer to cause the computer to perform: aprocess of acquiring load references of the respective touch panels of afirst operation display apparatus and a second operation displayapparatus among the plurality of operation display apparatuses, the loadreferences serving as a reference of a minimum pressing force that isnecessary for deciding that the respective corresponding touch panelsreceive an input operation, wherein it is determined that an inputoperation is received via any one of the touch panels only in a case inwhich a pressing force on said any one of the touch panels is equal toor greater than the corresponding load reference of said any one of thetouch panels, wherein the load references are respectively stored inadvance in a memory of the corresponding operation display apparatus,and wherein the load references are acquired from the correspondingmemories; a process of comparing the load reference of the firstoperation display apparatus with the load reference of the secondoperation display apparatus; and a process of, when an operation objecthas been displayed across a screen of the first operation displayapparatus and a screen of the second operation display apparatus,changing display control for displaying the operation object in responseto a result of the comparison.
 24. A non-transitory computer readablerecording medium storing an operation display program for controlling anoperation display system having a plurality of operation displayapparatuses each including a touch panel, wherein the operation displayprogram is executable by a computer to cause the computer to perform: aprocess of acquiring load references of the respective touch panels of afirst operation display apparatus and a second operation displayapparatus among the plurality of operation display apparatuses, the loadreferences serving as a reference of a minimum pressing force that isnecessary for deciding that the respective corresponding touch panelsreceive an input operation, wherein it is determined that an inputoperation is received via any one of the touch panels only in a case inwhich a pressing force on said any one of the touch panels is equal toor greater than the corresponding load reference of said any one of thetouch panels, wherein the load references are stored in advance in amemory of the corresponding operation display apparatus, and wherein theload references are acquired from the corresponding memories; a processof comparing the load reference of the first operation display apparatuswith the load reference of the second operation display apparatus; and aprocess of, in a state in which an operation object has been displayedacross a screen of the first operation display apparatus and a screen ofthe second operation display apparatus, when the operation object ofeach screen has been touched, outputting a notification indicatinginformation on an operation of at least one of the touch panels inresponse to a result of the comparison.
 25. A non-transitory computerreadable recording medium storing an operation display program forcontrolling an operation display apparatus including a touch panel,wherein the operation display program is executable by a computer tocause the computer to perform: a process of acquiring a first loadreference serving as a reference of pressing force by which the touchpanel receives an operation and a second load reference serving as areference of pressing force by which a touch panel of another operationdisplay apparatus operated by a user most recently receives anoperation; a process of comparing the first load reference and thesecond load reference with each other; and a process of, when it isdetermined that the user intends to subsequently operate the operationdisplay apparatus after having most recently operated said anotheroperation display apparatus and a result of the comparison indicatesthat there is a difference between the first and second load references,outputting a notification indicating a degree of a change in pressingforce for pressing the touch panel of the operation display apparatus.26. A non-transitory computer readable recording medium storing anoperation display program for controlling an operation display apparatusincluding a touch panel, wherein the operation display program isexecutable by a computer to cause the computer to perform: a process ofacquiring a first load reference serving as a reference of a minimumpressing force that is necessary for deciding that the touch panelreceives an input operation and a second load reference serving as areference of a minimum pressing force that is necessary for decidingthat a touch panel of another operation display apparatus receives aninput operation, wherein it is determined that an input operation isreceived via any one of the touch panels only in a case in which apressing force on said any one of the touch panels is equal to orgreater than the corresponding one of the first load reference and thesecond load reference, wherein the first load reference is stored inadvance in a memory of the operation display apparatus, wherein thesecond load reference is stored in advance in a memory of said anotheroperation display apparatus, and wherein the first load reference andthe second load reference are acquired from the corresponding memories;a process of comparing the first load reference and the second loadreference with each other; a process of, when an operation object hasbeen displayed across a screen of the operation display apparatus and ascreen of said another operation display apparatus, changing displaycontrol for displaying the operation object in response to a result ofthe comparison.
 27. A non-transitory computer readable recording mediumstoring an operation display program for controlling an operationdisplay apparatus including a touch panel, wherein the operation displayprogram is executable by a computer to cause the computer to perform: aprocess of acquiring a first load reference serving as a reference of aminimum pressing force that is necessary for deciding that the touchpanel receives an input operation and a second load reference serving asa reference of a minimum pressing force that is necessary for decidingthat a touch panel of another operation display apparatus receives aninput operation, wherein it is determined that an input operation isreceived via any one of the touch panels only in a case in which apressing force on said any one of the touch panels is equal to orgreater than the corresponding one of the first load reference and thesecond load reference, wherein the first load reference is stored inadvance in a memory of the operation display apparatus, wherein thesecond load reference is stored in advance in a memory of said anotheroperation display apparatus, and wherein the first load reference andthe second load reference are acquired from the corresponding memories;a process of comparing the first load reference and the second loadreference with each other; and a process of, in a state in which anoperation object has been displayed across a screen of the operationdisplay apparatus and a screen of said another operation displayapparatus, when the operation object of each screen has been touched,outputting a notification indicating information on an operation of atleast one of the touch panels in response to a result of the comparison.